<!DOCTYPE html>
<html lang="en-us" dir="ltr">
    <head><meta charset='utf-8'>
<meta name='viewport' content='width=device-width, initial-scale=1'><meta name='description' content='pyspark存储到数据库 Write data to MySQL table df.write.format(&amp;ldquo;jdbc&amp;rdquo;) .option(&amp;ldquo;url&amp;rdquo;, &amp;ldquo;jdbc:mysql://localhost:3306/test_db&amp;rdquo;) .option(&amp;ldquo;driver&amp;rdquo;, &amp;ldquo;com.mysql.jdbc.Driver&amp;rdquo;) .option(&amp;ldquo;dbtable&amp;rdquo;, &amp;ldquo;test_table&amp;rdquo;) .option(&amp;ldquo;user&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;password&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;inferSchema&amp;rdquo;, &amp;ldquo;true&amp;rdquo;)\ # 自动判断类型 .mode(&amp;ldquo;append&amp;rdquo;) .save()
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
Define schema using StructType schema = StructType([ StructField(&amp;ldquo;id&amp;rdquo;, IntegerType(), True), StructField(&amp;ldquo;value&amp;rdquo;, StringType(), True) ]) .schema(schema) \ # 特殊类型
pandas笔记 使用pandas中一直需要查文档, 在此记录
删除行或列 用法：DataFrame.drop(labels=None,axis=0, index=None, columns=None, inplace=False)
参数说明： labels 就是要删除的行列的名字，用列表给定 axis 默认为0，指删除行，因此删除columns时要指定axis=1； index 直接指定要删除的行 columns 直接指定要删除的列 inplace=False，默认该删除操作不改变原数据，而是返回一个执行删除操作后的新dataframe； inplace=True，则会直接在原数据上进行删除操作，删除后无法返回。
因此，删除行列有两种方式： 1）labels=None,axis=0 的组合 2）index或columns直接指定要删除的行或列'><title>pandas笔记</title>

<link rel='canonical' href='http://meruro.cn/p/pandas%E7%AC%94%E8%AE%B0/'>

<link rel="stylesheet" href="/scss/style.min.5a8e892f6fa14515e9065eee1f5d2e05302606ec7f11750b2fb95198f9cdcbfd.css"><meta property='og:title' content='pandas笔记'>
<meta property='og:description' content='pyspark存储到数据库 Write data to MySQL table df.write.format(&amp;ldquo;jdbc&amp;rdquo;) .option(&amp;ldquo;url&amp;rdquo;, &amp;ldquo;jdbc:mysql://localhost:3306/test_db&amp;rdquo;) .option(&amp;ldquo;driver&amp;rdquo;, &amp;ldquo;com.mysql.jdbc.Driver&amp;rdquo;) .option(&amp;ldquo;dbtable&amp;rdquo;, &amp;ldquo;test_table&amp;rdquo;) .option(&amp;ldquo;user&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;password&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;inferSchema&amp;rdquo;, &amp;ldquo;true&amp;rdquo;)\ # 自动判断类型 .mode(&amp;ldquo;append&amp;rdquo;) .save()
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
Define schema using StructType schema = StructType([ StructField(&amp;ldquo;id&amp;rdquo;, IntegerType(), True), StructField(&amp;ldquo;value&amp;rdquo;, StringType(), True) ]) .schema(schema) \ # 特殊类型
pandas笔记 使用pandas中一直需要查文档, 在此记录
删除行或列 用法：DataFrame.drop(labels=None,axis=0, index=None, columns=None, inplace=False)
参数说明： labels 就是要删除的行列的名字，用列表给定 axis 默认为0，指删除行，因此删除columns时要指定axis=1； index 直接指定要删除的行 columns 直接指定要删除的列 inplace=False，默认该删除操作不改变原数据，而是返回一个执行删除操作后的新dataframe； inplace=True，则会直接在原数据上进行删除操作，删除后无法返回。
因此，删除行列有两种方式： 1）labels=None,axis=0 的组合 2）index或columns直接指定要删除的行或列'>
<meta property='og:url' content='http://meruro.cn/p/pandas%E7%AC%94%E8%AE%B0/'>
<meta property='og:site_name' content='梅鲁梅鲁梅鲁梅~'>
<meta property='og:type' content='article'><meta property='article:section' content='Post' /><meta property='article:tag' content='代码' /><meta property='article:published_time' content='2018-11-18T22:00:07&#43;00:00'/><meta property='article:modified_time' content='2018-11-18T22:00:07&#43;00:00'/>
<meta name="twitter:title" content="pandas笔记">
<meta name="twitter:description" content="pyspark存储到数据库 Write data to MySQL table df.write.format(&amp;ldquo;jdbc&amp;rdquo;) .option(&amp;ldquo;url&amp;rdquo;, &amp;ldquo;jdbc:mysql://localhost:3306/test_db&amp;rdquo;) .option(&amp;ldquo;driver&amp;rdquo;, &amp;ldquo;com.mysql.jdbc.Driver&amp;rdquo;) .option(&amp;ldquo;dbtable&amp;rdquo;, &amp;ldquo;test_table&amp;rdquo;) .option(&amp;ldquo;user&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;password&amp;rdquo;, &amp;ldquo;hive&amp;rdquo;) .option(&amp;ldquo;inferSchema&amp;rdquo;, &amp;ldquo;true&amp;rdquo;)\ # 自动判断类型 .mode(&amp;ldquo;append&amp;rdquo;) .save()
from pyspark.sql.types import StructType, StructField, IntegerType, StringType
Define schema using StructType schema = StructType([ StructField(&amp;ldquo;id&amp;rdquo;, IntegerType(), True), StructField(&amp;ldquo;value&amp;rdquo;, StringType(), True) ]) .schema(schema) \ # 特殊类型
pandas笔记 使用pandas中一直需要查文档, 在此记录
删除行或列 用法：DataFrame.drop(labels=None,axis=0, index=None, columns=None, inplace=False)
参数说明： labels 就是要删除的行列的名字，用列表给定 axis 默认为0，指删除行，因此删除columns时要指定axis=1； index 直接指定要删除的行 columns 直接指定要删除的列 inplace=False，默认该删除操作不改变原数据，而是返回一个执行删除操作后的新dataframe； inplace=True，则会直接在原数据上进行删除操作，删除后无法返回。
因此，删除行列有两种方式： 1）labels=None,axis=0 的组合 2）index或columns直接指定要删除的行或列">
    </head>
    <body class="
    article-page
    ">
    <script>
        (function() {
            const colorSchemeKey = 'StackColorScheme';
            if(!localStorage.getItem(colorSchemeKey)){
                localStorage.setItem(colorSchemeKey, "auto");
            }
        })();
    </script><script>
    (function() {
        const colorSchemeKey = 'StackColorScheme';
        const colorSchemeItem = localStorage.getItem(colorSchemeKey);
        const supportDarkMode = window.matchMedia('(prefers-color-scheme: dark)').matches === true;

        if (colorSchemeItem == 'dark' || colorSchemeItem === 'auto' && supportDarkMode) {
            

            document.documentElement.dataset.scheme = 'dark';
        } else {
            document.documentElement.dataset.scheme = 'light';
        }
    })();
</script>
<div class="container main-container flex on-phone--column extended"><aside class="sidebar left-sidebar sticky ">
    <button class="hamburger hamburger--spin" type="button" id="toggle-menu" aria-label="Toggle Menu">
        <span class="hamburger-box">
            <span class="hamburger-inner"></span>
        </span>
    </button>

    <header>
        
            
            <figure class="site-avatar">
                <a href="/">
                
                    
                    
                    
                        
                        <img src="/img/ava_hu31add2dceb44736f969f74989490346b_19511_300x0_resize_q75_box.jpg" width="300"
                            height="300" class="site-logo" loading="lazy" alt="Avatar">
                    
                
                </a>
                
                    <span class="emoji">🍥</span>
                
            </figure>
            
        
        
        <div class="site-meta">
            <h1 class="site-name"><a href="/">梅鲁梅鲁梅鲁梅~</a></h1>
            <h2 class="site-description">朝闻道,夕死可矣.</h2>
        </div>
    </header><ol class="social-menu">
            
                <li>
                    <a 
                        href='https://github.com/CaiJimmy/hugo-theme-stack'
                        target="_blank"
                        title="GitHub"
                    >
                        
                        
                            <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-brand-github" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z" fill="none"/>
  <path d="M9 19c-4.3 1.4 -4.3 -2.5 -6 -3m12 5v-3.5c0 -1 .1 -1.4 -.5 -2c2.8 -.3 5.5 -1.4 5.5 -6a4.6 4.6 0 0 0 -1.3 -3.2a4.2 4.2 0 0 0 -.1 -3.2s-1.1 -.3 -3.5 1.3a12.3 12.3 0 0 0 -6.2 0c-2.4 -1.6 -3.5 -1.3 -3.5 -1.3a4.2 4.2 0 0 0 -.1 3.2a4.6 4.6 0 0 0 -1.3 3.2c0 4.6 2.7 5.7 5.5 6c-.6 .6 -.6 1.2 -.5 2v3.5" />
</svg>



                        
                    </a>
                </li>
            
                <li>
                    <a 
                        href='https://twitter.com'
                        target="_blank"
                        title="Twitter"
                    >
                        
                        
                            <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-brand-twitter" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z" fill="none"/>
  <path d="M22 4.01c-1 .49 -1.98 .689 -3 .99c-1.121 -1.265 -2.783 -1.335 -4.38 -.737s-2.643 2.06 -2.62 3.737v1c-3.245 .083 -6.135 -1.395 -8 -4c0 0 -4.182 7.433 4 11c-1.872 1.247 -3.739 2.088 -6 2c3.308 1.803 6.913 2.423 10.034 1.517c3.58 -1.04 6.522 -3.723 7.651 -7.742a13.84 13.84 0 0 0 .497 -3.753c-.002 -.249 1.51 -2.772 1.818 -4.013z" />
</svg>



                        
                    </a>
                </li>
            
        </ol><ol class="menu" id="main-menu">
        
        
        

        <li >
            <a href='/' >
                
                
                
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-home" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <polyline points="5 12 3 12 12 3 21 12 19 12" />
  <path d="M5 12v7a2 2 0 0 0 2 2h10a2 2 0 0 0 2 -2v-7" />
  <path d="M9 21v-6a2 2 0 0 1 2 -2h2a2 2 0 0 1 2 2v6" />
</svg>



                
                <span>Home</span>
            </a>
        </li>
        
        

        <li >
            <a href='/about/' >
                
                
                
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-user" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="12" cy="7" r="4" />
  <path d="M6 21v-2a4 4 0 0 1 4 -4h4a4 4 0 0 1 4 4v2" />
</svg>



                
                <span>About</span>
            </a>
        </li>
        
        

        <li >
            <a href='/archives/' >
                
                
                
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-archive" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <rect x="3" y="4" width="18" height="4" rx="2" />
  <path d="M5 8v10a2 2 0 0 0 2 2h10a2 2 0 0 0 2 -2v-10" />
  <line x1="10" y1="12" x2="14" y2="12" />
</svg>



                
                <span>Archives</span>
            </a>
        </li>
        
        

        <li >
            <a href='/search/' >
                
                
                
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-search" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="10" cy="10" r="7" />
  <line x1="21" y1="21" x2="15" y2="15" />
</svg>



                
                <span>Search</span>
            </a>
        </li>
        
        

        <li >
            <a href='/links/' >
                
                
                
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-link" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <path d="M10 14a3.5 3.5 0 0 0 5 0l4 -4a3.5 3.5 0 0 0 -5 -5l-.5 .5" />
  <path d="M14 10a3.5 3.5 0 0 0 -5 0l-4 4a3.5 3.5 0 0 0 5 5l.5 -.5" />
</svg>



                
                <span>Links</span>
            </a>
        </li>
        

        <div class="menu-bottom-section">
                <li id="i18n-switch">  
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-language" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z" fill="none"/>
  <path d="M4 5h7" />
  <path d="M9 3v2c0 4.418 -2.239 8 -5 8" />
  <path d="M5 9c-.003 2.144 2.952 3.908 6.7 4" />
  <path d="M12 20l4 -9l4 9" />
  <path d="M19.1 18h-6.2" />
</svg>



                    <select name="language" onchange="window.location.href = this.selectedOptions[0].value">
                        
                            <option value="http://meruro.cn/" selected>English</option>
                        
                            <option value="http://meruro.cn/zh-cn/" >中文</option>
                        
                            <option value="http://meruro.cn/ar/" >عربي</option>
                        
                    </select>
                </li>
            
            
            
                <li id="dark-mode-toggle">
                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-toggle-left" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="8" cy="12" r="2" />
  <rect x="2" y="6" width="20" height="12" rx="6" />
</svg>



                    <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-toggle-right" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="16" cy="12" r="2" />
  <rect x="2" y="6" width="20" height="12" rx="6" />
</svg>



                    <span>Dark Mode</span>
                </li>
            
        </div>
    </ol>
</aside>
<main class="main full-width">
    <article class="main-article">
    <header class="article-header">

    <div class="article-details">
    

    <div class="article-title-wrapper">
        <h2 class="article-title">
            <a href="/p/pandas%E7%AC%94%E8%AE%B0/">pandas笔记</a>
        </h2>
    
        
    </div>

    
    <footer class="article-time">
        
            <div>
                <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-calendar-time" width="56" height="56" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <path d="M11.795 21h-6.795a2 2 0 0 1 -2 -2v-12a2 2 0 0 1 2 -2h12a2 2 0 0 1 2 2v4" />
  <circle cx="18" cy="18" r="4" />
  <path d="M15 3v4" />
  <path d="M7 3v4" />
  <path d="M3 11h16" />
  <path d="M18 16.496v1.504l1 1" />
</svg>
                <time class="article-time--published">Nov 18, 2018</time>
            </div>
        

        
            <div>
                <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-clock" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="12" cy="12" r="9" />
  <polyline points="12 7 12 12 15 15" />
</svg>



                <time class="article-time--reading">
                    2 minute read
                </time>
            </div>
        
    </footer>
    

    
</div>
</header>

    <section class="article-content">
    
    
    <h2 id="pyspark存储到数据库">pyspark存储到数据库</h2>
<h1 id="write-data-to-mysql-table">Write data to MySQL table</h1>
<p>df.write.format(&ldquo;jdbc&rdquo;) <br>
.option(&ldquo;url&rdquo;, &ldquo;jdbc:mysql://localhost:3306/test_db&rdquo;) <br>
.option(&ldquo;driver&rdquo;, &ldquo;com.mysql.jdbc.Driver&rdquo;) <br>
.option(&ldquo;dbtable&rdquo;, &ldquo;test_table&rdquo;) <br>
.option(&ldquo;user&rdquo;, &ldquo;hive&rdquo;) <br>
.option(&ldquo;password&rdquo;, &ldquo;hive&rdquo;) <br>
.option(&ldquo;inferSchema&rdquo;, &ldquo;true&rdquo;)\ # 自动判断类型
.mode(&ldquo;append&rdquo;) <br>
.save()</p>
<p>from pyspark.sql.types import StructType, StructField, IntegerType, StringType</p>
<h2 id="define-schema-using-structtype">Define schema using StructType</h2>
<p>schema = StructType([
StructField(&ldquo;id&rdquo;, IntegerType(), True),
StructField(&ldquo;value&rdquo;, StringType(), True)
])
.schema(schema) \ # 特殊类型</p>
<h2 id="pandas笔记">pandas笔记</h2>
<p>使用pandas中一直需要查文档, 在此记录</p>
<h3 id="删除行或列">删除行或列</h3>
<p>用法：DataFrame.drop(labels=None,axis=0, index=None, columns=None, inplace=False)</p>
<p>参数说明：
labels 就是要删除的行列的名字，用列表给定
axis 默认为0，指删除行，因此删除columns时要指定axis=1；
index 直接指定要删除的行
columns 直接指定要删除的列
inplace=False，默认该删除操作不改变原数据，而是返回一个执行删除操作后的新dataframe；
inplace=True，则会直接在原数据上进行删除操作，删除后无法返回。</p>
<p>因此，删除行列有两种方式：
1）labels=None,axis=0 的组合
2）index或columns直接指定要删除的行或列</p>
<p>删除column一定要指定axis=1,否则会报错</p>
<h3 id="数据筛选">数据筛选</h3>
<pre><code>$ df.query('1&lt;label&lt;0')
</code></pre>
<h3 id="长度">长度</h3>
<pre><code>$ df.shape[0]
</code></pre>
<h3 id="dataframe">dataframe</h3>
<p>import pandas as pd 
import numpy as np
导入数据#默认utf-8，gbk对中文的支持更好
pd.read_csv(filename,encoding=&lsquo;gbk&rsquo;)</p>
<h1 id="key是列名value是数据从字典导入数据需要指定index">key是列名，value是数据,从字典导入数据需要指定index</h1>
<p>dict1 = {&lsquo;a&rsquo;:1,&lsquo;b&rsquo;:2}
df = pd.DataFrame(dict1,index=[0])查看数据#行数、列数
df.shape[0]
df.shape[1]</p>
<h1 id="前几行后几行默认为5">前几行，后几行，默认为5</h1>
<p>df.head()
df.tail(3)</p>
<h1 id="数据汇总统计">数据汇总统计</h1>
<p>df.describe()</p>
<h1 id="数据概况">数据概况</h1>
<p>df.info()</p>
<h1 id="列名">列名</h1>
<p>df.columns</p>
<h1 id="数据类型">数据类型</h1>
<p>df.dtypes</p>
<h1 id="各列平均值">各列平均值</h1>
<p>df.mean()数据选取#按列名查看某列
df[col_name]
df.col_name</p>
<h1 id="查列名看多列">查列名看多列</h1>
<p>df[[col1,col2]]</p>
<h1 id="按索引查看某行第一行">按索引查看某行,第一行：</h1>
<p>df.iloc[0]</p>
<h1 id="按索引查看某行某列三行四列那个元素">按索引查看某行某列，三行四列那个元素</h1>
<p>df.iloc[2,3]数据合并#按行拼接
pieces = [df[2:20],df[23:30],df[35:40]]下面这句replace知乎一直报未知错误（醉了，我在整篇文章中用二分法试出来两句），只能截图#按列拼接
left = pd.DataFrame({&lsquo;key&rsquo;: [&lsquo;foo&rsquo;, &lsquo;bar&rsquo;], &lsquo;col1&rsquo;: [1, 2]})
right = pd.DataFrame({&lsquo;key&rsquo;: [&lsquo;foo&rsquo;, &lsquo;bar&rsquo;], &lsquo;col2&rsquo;: [4, 5]})
pd.merge(left,right,on=&lsquo;key&rsquo;)</p>
<h1 id="增加一列">增加一列</h1>
<p>df = pd.DataFrame(np.random.randn(8, 4), columns=[&lsquo;A&rsquo;,&lsquo;B&rsquo;,&lsquo;C&rsquo;,&lsquo;D&rsquo;])
extra = df.iloc[-1]
data = df.append(extra)数据清理 (注意: 修改DataFrame要赋值)#检查是否空值，返回布尔值（True,False)
df.isnull()</p>
<h1 id="返回各列空值的个数">返回各列空值的个数</h1>
<p>df.isnull().sum()</p>
<h1 id="处理空值用x替换空值">处理空值，用x替换空值</h1>
<p>df = df.fillna(x)</p>
<h1 id="删除包含空值的行">删除包含空值的行</h1>
<p>df = df.dropna()</p>
<h1 id="删除包含空值的列">删除包含空值的列</h1>
<p>df = df.dropna(axis=1)数据处理#选择某列大于1的行
df[df[colname]&gt;1]</p>
<h1 id="选择某列包含x或者y字段的行pandas有很多字符串处理函数">选择某列包含x或者y字段的行,pandas有很多字符串处理函数</h1>
<p>df[df[colname].str.contains(x|y)]下面这句replace知乎一直报未知错误（再次醉了），只能截图#转换数据类型
df.num = df.num.astype(float)</p>
<h1 id="查看某列的唯一值的个数">查看某列的唯一值的个数</h1>
<p>df.colname.value_counts()</p>
<h1 id="按照某列排序默认升序">按照某列排序（默认升序）</h1>
<p>df.sort_values(by = colname,ascending=True)</p>
<h1 id="应用函数简单的可以用lambda">应用函数，简单的可以用lambda</h1>
<p>df.apply(lambda x:x.max()-x.min())</p>
<h1 id="复杂的可以先定义好函数">复杂的可以先定义好函数</h1>
<p>def func():
pass
df.apply(func)</p>
<h1 id="也可以用numpy自带的例如cumsum累加">也可以用numpy自带的，例如cumsum累加</h1>
<p>df.apply(np.cumsum)分组聚合#准备数据
df = pd.DataFrame({&lsquo;A&rsquo; : [&lsquo;foo&rsquo;, &lsquo;bar&rsquo;, &lsquo;foo&rsquo;, &lsquo;bar&rsquo;,
&lsquo;foo&rsquo;, &lsquo;bar&rsquo;, &lsquo;foo&rsquo;, &lsquo;foo&rsquo;],
&lsquo;B&rsquo; : [&lsquo;one&rsquo;, &lsquo;one&rsquo;, &rsquo;two&rsquo;, &rsquo;three&rsquo;,
&rsquo;two&rsquo;, &rsquo;two&rsquo;, &lsquo;one&rsquo;, &rsquo;three&rsquo;],
&lsquo;C&rsquo; : np.random.randn(8),
&lsquo;D&rsquo; : np.random.randn(8)})</p>
<h1 id="对两列进行分组并求均值类似于sql里的group-by">对两列进行分组并求均值，类似于SQL里的group by</h1>
<p>df.groupby([&lsquo;A&rsquo;,&lsquo;B&rsquo;]).mean()</p>
<h1 id="数据透视表以col1为行col2为列取col3和col4的最大值可以参考excel的数据透视表">数据透视表，以col1为行，col2为列，取col3和col4的最大值，可以参考Excel的数据透视表</h1>
<p>df.pivot_table(index=col1,columns=col2,values=[col3,col4],aggfunc=max)</p>
<h1 id="使用lambda函数">使用lambda函数</h1>
<p>apply(lambda x:&ldquo;<a class="link" href="https://www.douyin.com/user/%22&#43;x"  target="_blank" rel="noopener"
    >https://www.douyin.com/user/"+x</a>) 
参数表达式,自动返回表达式的结果</p>

</section>


    <footer class="article-footer">
    
    <section class="article-tags">
        
            <a href="/tags/%E4%BB%A3%E7%A0%81/">代码</a>
        
    </section>


    
    <section class="article-copyright">
        <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-copyright" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <circle cx="12" cy="12" r="9" />
  <path d="M14.5 9a3.5 4 0 1 0 0 6" />
</svg>



        <span>Licensed under CC BY-NC-SA 4.0</span>
    </section>
    </footer>


    
</article>

    

    

     
    
        
    

    <footer class="site-footer">
    <section class="copyright">
        &copy; 
        
            2020 - 
        
        2023 梅鲁梅鲁梅鲁梅~
    </section>
    
    <section class="powerby">
        Built with <a href="https://gohugo.io/" target="_blank" rel="noopener">Hugo</a> <br />
        Theme <b><a href="https://github.com/CaiJimmy/hugo-theme-stack" target="_blank" rel="noopener" data-version="3.13.0">Stack</a></b> designed by <a href="https://jimmycai.com" target="_blank" rel="noopener">Jimmy</a>
    </section>
</footer>


    
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    
    <div class="pswp__bg"></div>

    
    <div class="pswp__scroll-wrap">

        
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                
                
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div><script 
                src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"integrity="sha256-ePwmChbbvXbsO02lbM3HoHbSHTHFAeChekF1xKJdleo="crossorigin="anonymous"
                defer
                >
            </script><script 
                src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"integrity="sha256-UKkzOn/w1mBxRmLLGrSeyB4e1xbrp4xylgAWb3M42pU="crossorigin="anonymous"
                defer
                >
            </script><link 
                rel="stylesheet" 
                href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css"crossorigin="anonymous"
            ><link 
                rel="stylesheet" 
                href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css"crossorigin="anonymous"
            >

            </main>
    <aside class="sidebar right-sidebar sticky">
        
            
                
    <section class="widget archives">
        <div class="widget-icon">
            <svg xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-hash" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentColor" fill="none" stroke-linecap="round" stroke-linejoin="round">
  <path stroke="none" d="M0 0h24v24H0z"/>
  <line x1="5" y1="9" x2="19" y2="9" />
  <line x1="5" y1="15" x2="19" y2="15" />
  <line x1="11" y1="4" x2="7" y2="20" />
  <line x1="17" y1="4" x2="13" y2="20" />
</svg>



        </div>
        <h2 class="widget-title section-title">Table of contents</h2>
        
        <div class="widget--toc">
            <nav id="TableOfContents">
  <ol>
    <li><a href="#pyspark存储到数据库">pyspark存储到数据库</a></li>
  </ol>

  <ol>
    <li><a href="#define-schema-using-structtype">Define schema using StructType</a></li>
    <li><a href="#pandas笔记">pandas笔记</a>
      <ol>
        <li><a href="#删除行或列">删除行或列</a></li>
        <li><a href="#数据筛选">数据筛选</a></li>
        <li><a href="#长度">长度</a></li>
        <li><a href="#dataframe">dataframe</a></li>
      </ol>
    </li>
  </ol>
</nav>
        </div>
    </section>

            
        
    </aside>


        </div>
        <script 
                src="https://cdn.jsdelivr.net/npm/node-vibrant@3.1.6/dist/vibrant.min.js"integrity="sha256-awcR2jno4kI5X0zL8ex0vi2z&#43;KMkF24hUW8WePSA9HM="crossorigin="anonymous"
                
                >
            </script><script type="text/javascript" src="/ts/main.js" defer></script>
<script>
    (function () {
        const customFont = document.createElement('link');
        customFont.href = "https://fonts.googleapis.com/css2?family=Lato:wght@300;400;700&display=swap";

        customFont.type = "text/css";
        customFont.rel = "stylesheet";

        document.head.appendChild(customFont);
    }());
</script>

    </body>
</html>
